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Abstract — In this study , we propose an interaction method for 3D voxel based modeling in immersive virtual environments. 
We assumed that all tasks must be NUI (Natural User Interface ) based interactions, rather than using conventional input 
devices like keyboard or mouse. To see the usefulness of our proposed interaction methods, we have created an immersive 
virtual reality application that can interact with three-dimensional objects using only hands and arms. The modeling 
process requires interactions for two types of tasks. One group is the menu selection and control tasks, and the other is the 
manipulation tasks for voxels. We have defined motion interactions that are appropriate for each task so that all actions are 
possible only with hands and arms. In order to confirm how effective the proposed methods are, we have produced various 
shapes of 3d objects. Based on the results, we investigated the possibility of NUI based interactions proposed in this study. 

Keywords — NUI ( Natural User Interface), HMD (Head Mounted Display), immersive, 3d modeling. 

I. Introduction 

With the advent of Oculus Rif in 2012, which uses advanced technology, and the introduction of various virtual reality 
contents using it, HMD-based immersive virtual reality has received great interest from people. On the other hand, the 
development of input devices that interact with virtual reality is relatively lagging behind the development of HMD. The 
contents that appeared in the early days used conventional devices such as keyboards, mice, and game controllers as the 
interaction devices between users and virtual reality. However, these input devices act as obstacles to the user's immersion 
into the virtual reality. For this reason, recently, devices that allow more natural interactions are being announced, and in a 
near future, virtual reality that does not require cumbersome devices is expected to become popular. As a result, if users are 
free to interact with both arms, the user will feel more immersive and the performance of tasks in virtual environments will 
be much higher. 

Just as it is quite natural nowadays, the GUI (Graphical User Interface) we are using has evolved from the former CLI 
(Command Line Interface) to the interactive environment using WIMP (Windows, Icon, Menu, Pointer). Moreover, in recent 
years, the transition to the NUI (Natural User Interface) environment has been progressing gradually due to the development 
of various virtual reality related technologies. We started this study to prepare for NUI -based computing environment to be 
unfolded in a near future. 

The structure of this paper is as follows. In Chapter 2, we describe the HMD as an immersive V R input device and the NUI- 
based interactions that are most suitable for a virtual environment. In Chapter 3, we introduce the researches related to this 
study. In Chapter 4, we describe the HW configuration and software functions of our 3 -dimensional voxel -based modeling 
tool. Chapter 5 explains an example modeling process, and Chapter 6 concludes. 

II. Research Background 
2.1 An immersive VR input device: HMD 

An immersive device, an HMD, is a type of a display device worn on the head. The user sees images output from two small 
display panels in front of his eyes. In the past, HMDs used expensive lenses to create realistic screens, the viewing angle was 
as narrow as 30 to 45 degrees, and the equipment was heavy. As a result, the price was very high, and immersion and 
realism were not great. However, in recent years, various technologies related to the HMD have been developed to overcome 
these problems. 

One of those devices, Oculus rift, was designed and built by Palmer Luckey. Oculus rift uses distorted images to show wide 
viewing angles from 95 to 110 degrees. These images are distorted by compressing original images with wide viewing 
angles, and when viewed through fisheye lenses, distorted images are corrected and wide screens can be seen. In addition, a 
gyro sensor, an accelerometer, and a magnetometer are also embedded at a very low price compared with the past. Since 
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then, a variety of popular HMDs have been released, making immersive virtual environments more accessible to the general 
public, not just professionals. 

2.2 User Interface for virtual reality: HMD 

When computers first appeared in the world, the way they used computers was through a batch process. Punch cards were 
used as an input device, and the output device was a line printer only. At this time, however, the term “user interface” was 
rarely used. The beginning of the real user interface is the CLI (Command Line Interface). The CLI is an environment such 
as Unix Shell or DOS that responds to commands in text format that must be entered correctly in the system. Since the real- 
time 2D graphics could be processed and the mouse become popular, GUI-based operating system using WIMP (Windows, 
Icon, Menu, Pointer) become most. In recent years, however, due to the development of GPU technology, not only 2D but 
also high-quality 3D graphics can be processed in real time, and with the popularity of HMD, the general public is now able 
to access virtual reality technologies easily. In this immersive virtual environment, the conventional keyboard and mouse 
based GUI is not only not able to utilize the advantages of virtual reality but also become a stumbling block to users. Thus, 
many researchers have recently started to study NUI, a new interactive method. 

NUI refers a user interface interacting only with “Natural and Intuitive” actions, rather than indirectly interacting using 
physical devices. A typical example is interaction through action or voice. The NUI has contact-type methods of operating 
the device directly in contact with the body, and non-contact methods of operating the device based on recognition 
technology without direct contact with the body. The contact type includes multi-touch, sensor recognition, brain wave 
recognition, and non-contact type includes speech recognition and motion recognition. 

Multi-touch, which is a typical example of the contact type, is a technique of recognizing a plurality of touch points at the 
same time, and can perform more various operations than one point recognition. Nowadays, popularized smart phones are 
capable of flipping, rotating, and zooming operations by using multi touch. 

The voice recognition interface, which is one of the non-contact type, is a technology for converting a user's speech into text, 
analyzing and processing the text. It is developing towards recognizing many vocabulary and natural conversation for all 
users and increasing recognition rate. Another non-contact approach, motion recognition, recognizes and tracks the user's 
movement. It replaces the role of the conventional mouse or joystick by utilizing body parts such as hands, arms, and legs. 
In order to compensate for the problem of occlusion and light interference, which is one of the disadvantages of the 
approaches using cameras, it is possible to use 2 or more cameras or to compensate date with additional sensors such as gyro 
sensors and accelerometers. 


III. Related Works 

Recently, as immersive virtual reality devices such as HMD have become popularized, the public interest in NUI has 
increased very much, but research on NUI has already started for a long time. Among them, there are many researches for 
utilizing virtual reality or augmented reality contents in various fields. The application areas are so diverse that NUI research 
has been done for areas such as medicine [1], architecture [2], and VR for the elderly [3]. 

The console game makers, such as Sony, Nintendo, and Microsoft, have also actively conducted research on sensor devices 
for NUI and sold sensor devices for games. In [4], the author presents a review of the NUI, and comparisons and analysis 
results of these sensor devices released by those companies. NUI devices for a variety of applications other than gaming are 
also being released in recent years. Among them, Leap Motion is widely used in many research institutes including this 
research. Examples include manipulating complex data in point cloud with a natural motion called "Slice-n-Swipe" [5] and 
interacting without touching objects by hands in dental surgery [6]. 

Similar to this study, there are many studies for modeling 3D objects in virtual space by NUI based interaction. A study for 
interactive sketching in virtual space [7], research for design and manufacturing in VR/AR space [8], and research for 
prototyping architecture [9] can be found. However, our approach is for the voxel-based modeling, and there are differences 
in interaction method with the above studies. Voxels were originally designed to simplify very complex objects. As can be 
seen in [10], the voxel was used to model complex tree shapes. 

In [11], the author describes “7 typical characteristics” of NUI. The characteristics include “user-centered,” “multi-channel,” 
“inexact,” “high bandwidth,” “voice-based interaction,” “image-based interaction,” and “behavior-based interaction.” As 
mentioned in this paper, as technology develops day by day, it is expected that NUI will not inexact and require high 
bandwidth in a near future. 
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IV. 3D Voxel-based Modeling Tool 

The application developed in this study is a tool that can model 3D objects using voxels only by the motion of hands and 
arms in immersive virtual environment. This chapter describes the HW configuration of the system, SW functions for menu 
and voxel manipulation, and the interactions used. 

4.1 Hardware Configuration 


We used Oculus DK2 as an output device, and used Leap Motion and Myo as sensor devices for user's motion recognition. 
Unity Game engine was used as SW development tool. As shown in Fig. 1, Leap Motion sensor was attached to the HMD 
and 2 Myos were attached to both arms. 



Fig. 1: Hardware Configuration of our system (left: Hmd with Myo, center: Leap Motion, right: Myo) 


Lip Motion, developed by a venture company in San Francisco, is a sensor that can recognize hand movements precisely, and 
provides various interaction methods on a desktop PC. Despite its small size of 76mm x 30mm x 12.7mm, it is able to 
recognize the minute movement of fingers in 8 cubic feet of space. The recognition radius is 150 degrees, and small 
movement of 0.01mm can be detected. Since the motion is tracked at a speed of 200 FPS, the latency is also very small. 

Myo, developed by Thalmic Labs, is an armband-type motion recognition controller. Myo consists of a 3 -axis gyroscope, a 
3-axis magnetometer, a 3-axis accelerometer, a 9-axis inertial measurement device, and an EMG(electromyography) sensor 
that responds to the movement of 8 muscles. This device measures and senses the movement of the muscles in an arm and 
recognizes what user’s hands and arms are doing. In addition, it can act as a presentation controller, an RC device 
manipulator, and a game controller. 

4.2 Software Functions and Interactions 

The main menu of the proposed Modeling Tool consists of 6 items [Fig. 2 left]. They include “Save,” “Load,” “Clear,” 
“Colors,” “Pictures,” “Manipulate.” In addition, a button for fixing the menu is located at the upper right of the menu plate. 
In order to display the main menu in a virtual space, you can pinch with the thumb and index finger of your left hand, and as 
your hand moves, the menu moves along. In order to select a menu item, touch action is required with the index finger of the 
right hand. 

“Colors” sub-menu allows you to choose the color of the voxel to be created [Fig. 2 center]. There are located three slider 
bars in this sub-menu. You can change the color by touching and dragging each slider bar with your finger. This sub-menu 
disappears when you select another menu item from the main menu or select the "Colors" menu again. As with the main 
menu, all the sub-menus have a “fix” button, so that each sub-menu can be placed at a desired position. Selecting “Pictures” 
menu shows pictures that can be used as a reference for modeling. Due to the nature of immersive virtual reality, it's a menu 
made because you cannot see the outside of HMD. “Manipulate” sub-menu allows you to move, reduce or enlarge, and 
rotate the desired object. To move, select the desired object with the right hand pinch motion and move it. To reduce or 
enlarge, you should pinch with both hands and then open or narrow both hands. Finally, in order to rotate, you can move 
your right hand without moving your left hand. 
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Fig. 2: Menu Operations (left: Main Menu, center: Colors Menu, right: Pictures Menu) 


When you create a voxel, if the exact position and the color of the block cannot be confirmed, you will experience many 
trials and errors. Therefore, we implemented the function to preview the voxel to be generated. You can see the preview 
voxel by moving the index finger to the position you want to create. However, it is possible to preview only if there is no 
voxel created beforehand in the position of the index finger. Once you have confirmed that the preview voxel is what you 
want to create, the voxel creation process is completed with the double -tap action of the left hand. The deletion operation of 
the voxel is similar to the creation. Once you choose the voxel you want to delete with your right index finger, you can 
delete it with the double-tap action of your left hand. 


V. An Example Modeling Process 



Fig. 3: Screenshots of Modeling Process 


We have created various models to see how useful the tool and user interface. This chapter briefly describes the process by 
which one model was designed. First, we used left hand to call up the main menu, and called up “Pictures” sub-menu and 
selected one of the pictures by right hand. Then, we used “Manipulate” sub-menu to move to a position where the floor is 
well visible. We called up “Colors” sub-menu to change the color of the. Once the desired color is selected, we moved the 
index finger of right hand to the position we want to create and then we double-tapped with left hand. We repeated this 
operation and felt the fatigue of left hand and fixed the main menu in a well visible place. Then, in order to make another 
face, we selected “Manipulate” sum-menu and rotated the object under construction. When the fixed menu was overlapped, 
we moved the menu to another position by right hand. Occasionally, a fixed menu window became too small to be visible. 
Then, we could enlarge it by using both hands while viewing the menu. 

IV. Conclusion 

In this study, we investigated the NUI-based interaction. To do this, we developed a 3D voxel-based modeling tool for 
immersive virtual environments using HMDs, and modeled various 3D objects using the developed tools. 


Page | 12 





International Journal of Engineering Research & Science (IJOER) 


ISSN: [2395-6992] 


[Vol-3, Issue-2, February- 2017] 


As a result of modeling, most of the operations were able to be used without difficulty in the virtual space. A typical 
example is pinch operation. In the case of pinch, it was used for menu call and voxel movement and rotation. The double- 
tapping action of creating a voxel is not used very well in real life, but since it is a frequently used command, the simple 
operation seemed rather comfortable. In the case of the touch operation used to select the menu, since it is a frequently used 
operation in our real life in order to press the screen on a smartphone, it was possible to naturally perform the action. The 
pinch and drag action used to move the fixed menu could also be performed naturally. 

On the other hand, there were also some problems that were pointed out as disadvantages. When we designed a complex 
object, we had to do the same operation repeatedly many times. When designing by NUI-based interactions, we can see that 
natural behavior is not always an advantage, although we feel very natural. It is very important that interaction behaviors 
should be natural but not physical fatigue. There was also the opinion that in the case of the touch operation, the user feels a 
sense of heterogeneity. This is caused by the fact that we cannot feel physical feedback like smartphones or touch-screens. In 
recent years, devices with haptic capabilities that can provide physical feedback have also begun to be released, and this 
problem is expected to be solved in a near future. 
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